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Abstract 

We present several new characterizations of correlated equilibria in games with 
continuous utility functions. These have the advantage of being more computationally 
and analytically tractable than the standard definition in terms of departure functions. 
We use these characterizations to construct effective algorithms for approximating a 
single correlated equilibrium or the entire set of correlated equilibria of a game with 
polynomial utility functions. 

1 Introduction 

In finite games correlated equilibria are simpler than Nash equilibria in several senses - 
mathematically at least, if not conceptually. The set of correlated equilibria is a convex 
polytope, described by finitely many explicit linear inequalities, while the set of (mixed) 
Nash equilibria need not be convex or connected and can contain components which look 
like essentially any real algebraic variety (set described by polynomial equations on real 
variables) [7|. The existence of correlated equilibria can be proven by elementary means 
(linear programming or game theoretic duality [15] ) , whereas the existence of Nash equilibria 
seems to require nonconstructive methods (e.g., fixed point theorems as in [22| IH]) or the 
analysis of complicated algorithms [HI [TOl [12] . Computing a sample correlated equilibrium 
or a correlated equilibrium optimizing some quantity such as social welfare can be done 
efficiently [131 E3]; strong evidence in complexity theory suggests that the corresponding 
problems for Nash equilibria are hard [HI HI [13] . 

There are several exceptional classes of games for which the above problems about Nash 
equilibria become easy. The most important here are the zero-sum games. Broadly speaking, 
Nash equilibria of these games have complexity similar to correlated equilibria of general 
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Table 1: Comparison of the simplest known description of different classes of equilibrium 
sets in finite and polynomial games. 

games. In particular, the set of Nash equilibria is an easily described convex polytope, 
existence can be proven by duality, and a sample equilibrium can be computed efficiently. 

The situation in games with infinite strategy sets is not nearly so clear. For the compu- 
tational sections of this paper we restrict attention to the simplest such class of games, those 
with finitely many players, strategy sets equal to [—1,1], and polynomial utility functions. 
We make this restriction for several reasons. 

The first is conceptual and notational simplicity. Results similar to ours will hold when 
the strategy sets are general compact semialgebraic (described by finitely many polynomial 
equations and inequalities) subsets of M". However, dealing with this additional level of 
generality requires machinery from computational real algebraic geometry and does little to 
illuminate our basic methods. 

The second is generality. Much of the study of games with infinite strategy sets is fraught 
with assumptions of concavity or quasiconcavity which appear to be motivated not by natural 
game theoretic premises, but rather by the inadequacy of available tools for games without 
these properties. While polynomiality assumptions and concavity assumptions are both rigid 
in their own ways, polynomials have the benefit of being dense in the space of all continuous 
functions, and thus suitable for approximating a much wider class of games. 

The third reason is convenience. The algebraic structure we gain by restricting atten- 
tion to polynomials allows us to use recent advances in semidefinite programming and real 
algebraic geometry to construct efficient algorithms and gain conceptual insights. 

Little is known about correlated equilibria of these polynomial games, but much is known 
about Nash equilibria. Most importantly, the set of mixed Nash equilibria is nonempty 
and admits a finite-dimensional description in terms of the moments of the players' mixed 
strategies [3T] . 

This set of moments can be described explicitly in terms of polynomial equations and 
inequalities j3l] • The Nash equilibrium conditions are expressible via first order statements, 
so the set of all moments of Nash equilibria is a real algebraic variety and can be computed 
in theory, albeit not efficiently in general. In the two-player zero-sum case, the set of Nash 
equilibria can be described by a semidefinite program (an SDP is a generalization of a linear 
program which can be efficiently solved; see the appendix), hence we can compute a sample 
Nash equilibrium or one which optimizes some linear functional in polynomial time [25]. A 
summary of the results described so far is shown in Table [1} 



2 



Contributions The impetus for this paper was to address the bottom right cell of Table 
[!} the one with the question mark. The table seems to suggest that the set of correlated 
equilibria of a polynomial game should be describable by a semidefinite program. We will see 
that this is approximately true, but not exactly. The contribution of this paper is twofold. 

• First, we present several new characterizations of correlated equilibria in games with 
continuous utility functions (polynomiality is not needed here). In particular we show 
that the standard definition of correlated equilibria in terms of measurable departure 
functions is equivalent to other definitions in which the utilities are integrated against 



all test functions in some class (Theorem 2.10). This characterization does not have 



any obvious game theoretic significance, but it is extremely useful analytically and it 
forms the base for our other contributions. 

Second, we present several algorithms for approximating correlated equilibria within 
an arbitrary degree of accuracy. We present one inefficient linear programming based 
method as a benchmark, followed by two semidefinite programming based algorithms 
which perform much better in practice. The first SDP algorithm, called adaptive 
discretization, iteratively computes a sequence of approximate correlated equilibria 



supported on finite sets (Section |3^). We enlarge the support sets at each iteration 
using a heuristic which guarantees convergence in general and yields fast convergence 
in practice. The second SDP algorithm, called moment relaxation, does not discretize 
the strategy spaces but instead works in terms of joint moments. It produces a nested 
sequence of outer approximations to the set of joint moments of correlated equilibrium 
distributions, and these approximate equilibrium sets are described by semidefinite 



programs (Section 3.3). These relaxations depend crucially on one of the correlated 



equilibrium characterizations we have developed. 

Related literature The questions we address and the techniques we use are inspired by 
existing literature in two main areas. First, our work is related to a number of papers in the 
game theory literature. 

• Aumann defines correlated equilibria in his famous paper [T] , focusing on finite games 
to establish the basic properties and important examples. He obtains existence as a 
consequence of Nash's theorem on the existence of Nash equilibria in finite games [22j . 

• Hart and Schmeidler show that existence of correlated equilibria in finite games can be 
proven directly by a duality argument [15] . They then use a careful limiting argument 
to prove existence of correlated equilibria in continuous games with compact Hausdorff 
strategy spaces (Theorem 3 of that paper). The germs of ideas in this limiting argument 
are developed further in Section[2]of the present paper to yield various characterizations 
of correlated equilibria. It is worth noting that in [15] the authors also consider part 
IQ of Corollary [2l4 candidate definition of correlated equilibria. They discard it 



is as not obviously capturing the game theoretic idea of correlated equilibrium, but we 
prove that it is nonetheless an equivalent definition in games with continuous utilities. 
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Stoltz and Lugosi study learning algorithms which converge to correlated equilibria 
in continuous games [33]. These algorithms have a game theoretic interpretation as 
avoiding "regret" in a repeated game setting. Each player can carry out these proce- 
dures separately without knowledge of his opponents' utilities. These are conceptual 
advantages over our methods, which merely aim for efficient computation. 

However, these advantages come at a cost. The learning procedures require each player 
to solve a fixed point equation at each iteration. In general finding fixed points is as 
hard as finding Nash equilibria [HI E] , so these procedures do not seem to lead directly 
to efficient methods for computing correlated equilibria of continuous games. 

There exist classes of fixed point equations which can be solved efficiently (i.e., the 
steady-state distribution of a Markov chain, which is defined by a linear program). To 
our knowledge there has been no work on whether the equations of [33] fall into such 
a class. 

Furthermore, each of these learning algorithms either makes concavity-type assump- 
tions about the utility functions, which we seek to avoid for modeling fiexibility, or 



discretizes the players' strategy spaces a priori. We will see in Section 3.1 that such 
discretization without regard to the structure of the game can result in slow conver- 
gence. 

However, we will make use of some of the tools which Stoltz and Lugosi have created. 
In particular, they consider replacing the class of all measurable departure functions 
with a smaller class, such as simple or continuous departure functions, and study when 
this yields an equivalent equilibrium notion. One result of this type is stated as Lemma 



2.11 below and used to prove our characterization theorems. 



Germano and Lugosi prove the existence of correlated equilibria with small support 
in finite games [TT]. To prove this they analyze the extreme points of the set of 
correlated equilibria. Such an analysis cannot carry over directly to polynomial games 
because the set of correlated equilibria of polynomial games may have extreme points 
with arbitrarily large finite support or with infinite support |32j. Support bounds 
for correlated equilibria of polynomial games are proven in [30] using similar tools, 
but assuming a finitely supported Nash equilibrium is on hand as a starting point. 
Since Nash equilibria are generally assumed to be harder to compute than correlated 
equilibria, these results do not apply in the present setting where the goal is efficient 
computation. 

Separately from the literature on correlated equilibria, Dresher, Karlin, and Shapley 
study the structure of Nash equilibria in zero-sum games with polynomial or sepa- 
rable (polynomial-like) utility functions. They show how to cast separable games as 
finite-dimensional "convex games" by replacing the infinite-dimensional mixed strategy 
spaces with finite-dimensional spaces of moments ^ and prove existence of equilibria 
via fixed point arguments [8]. There always exist finitely supported equilibria in sep- 
arable games as can be shown using the finite-dimensonality of the moment spaces. 
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The rich geometry of these spaces is studied in [T7] . Most of these results as well as ad 
hoc methods for computing equilibria in simple cases are summarized in Karlin's book 
|16j . The authors of the present paper study generalizations and extensions of these 
results to nonzero-sum separable games in [31j . 




Second, our work is related to results from the optimization and computer science liter- 
ature. 

• Aumann showed that the set of correlated equilibria of a finite game is defined by 
polynomially many (in the size of the payoff tables) linear inequalities [2] . However, it 
was not clear whether this meant they could be computed in polynomial time. This 
question was settled in the affirmative when Khachian proved that linear programs 
could be solved in polynomial time; for an overview of this and other more efficient 
algorithms, see [3J. Papadimitriou extends this result in [23], showing that correlated 
equilibria can be computed efficiently in many classes of games for which the payoffs 
can be written succinctly, even if the explicit payoff tables would be exponential in 
size. 

• The breakthrough in optimization most directly related to the work in this paper is 
the development of semidefinite programming, a far-reaching generalization of linear 
programming which is still polynomial-time solvable (for an overview, see the appendix 
and [35]). More specifically, the development of sum of squares methods has allowed 
many optimization problems involving polynomials or moments of measures to be 
solved efficiently [24j. Parrilo applies these techniques to efficiently compute Nash 
equilibria of two-player zero-sum polynomial games in [25J. 

The remainder of this paper is organized as follows. In Section [2] we define the classes 
of games we study and correlated equilibria thereof, then prove several characterization 
theorems. We present algorithms for approximating sample correlated equilibria and the set 
of correlated equilibria of polynomial games in Section |3} Finally, we close with conclusions 
and directions for future work. 

2 Characterizations of Correlated Equilibria 

In this section we will define finite and continuous games along with correlated equilibria 
thereof. We will present several known characterizations of correlated equilibria in finite 
games and show how these naturally extend to continuous games. 

Some notational conventions used throughout are that subscripts refer to players, while 
superscripts are frequently used for other indices (it will be clear from the context when they 
represent exponents). If Sj are sets for j = 1,. . . ,n then S = U^^^Sj and S'_j = Uj^iSj. 
The n-tuple s and the {n — l)-tuple s_j are formed from the points Sj similarly. The set of 
regular Borel probability measures vr over a compact Hausdorff space S is denoted by A(S'). 
For simplicity we will write 7r(s) in place of vr({s}) for the measure of a singleton {s} C S. 
All polynomials will be assumed to have real coefficients. 
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2.1 Finite Games 

We start with the definition of a finite game. 

Definition 2.1. A finite game consists of players i = 1, . . . ,n, each of whom has a finite 
pure strategy set Cj and a utility or payoff function : C — )■ M, where C = ITJ^iCj. 

Each player's objective is to maximize his (expected) utility. We now consider what it 
would mean for the players to maximize their utility if their strategy choices were correlated. 
Let i? be a random variable taking values in C distributed according to some measure 
71 G A(C). A realization of i? is a pure strategy profile (a choice of pure strategy for 
each player) and the i^^ component of the realization Ri will be called the recommendation 
to player i. Given such a recommendation, player i can use conditional probability to form 
a posteriori beliefs about the recommendations given to the other players. A distribution 
IT is defined to be a correlated equilibrium if no player can ever expect to unilaterally gain 
by deviating from his recommendation, assuming the other players play according to their 
recommendat ions . 

Definition 2.2. A correlated equilibrium of a finite game is a joint probability measure 
71 G A(C) such that if i? is a random variable distributed according to vr then 

E[ui{ti,R_i) -Ui{R)\Ri = Si\= ^ Prob(i? = s\Ri = Si) [ui{ti,S-i) - Ui{s)] < 

for all players i, all Si G Ci such that Prob(-Rj = Sj) > 0, and all ti E Ci. 

While this definition captures the idea we have described above, the following character- 
ization is easier to apply and visualize. 

Proposition 2.3. A joint probability measure vr G A(C) is a correlated equilibrium of a 
finite game if and only if 

7r{s)[ui{ti,s-i) -Ui{s)] <0 (1) 

for all players i and all Si,ti G Ci. 

This proposition shows that the set of correlated equilibria is defined by a finite number 
of linear equations and inequalities (those in ([T]) along with 7r(s) > for all s G C and 
^^g^7r(s) = 1) and is therefore convex and even polyhedral. It can be shown via linear 
programming duality that this set is nonempty [13]. This can be shown alternatively by 
appealing to the fact that Nash equilibria exist and are the same as correlated equilibria 
which are product distributions. 

We can think of correlated equilibria as joint distributions corresponding to recommen- 
dations which will be given to the players as part of an extended game. The players are then 
free to play any function of their recommendation as their strategy in the game. 
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Definition 2.4. A function Q : Ci ^ Ci is called a departure function. 

If it is a Nash equilibrium of this extended game for each player to play his recommended 
strategy (i.e. if no player has an incentive to unilaterally deviate from using the identity 
departure function), then the distribution is a correlated equilibrium. This interpretation is 
due to Aumann [2] and is justified by the following alternative characterization of correlated 
equilibria. 

Proposition 2.5. A joint probability measure tt G A(C) is a correlated equilibrium of a 
finite game if and only if 

J2 ^(S) [u^iCi{s^), S-i) - Ui{s)] < (2) 

sec 

for all players i and all departure functions Q. 

For examples and more discussion of the basics of correlated equilibria, including the 
ideas behind the equivalence of these characterizations, see [HE]- 



2.2 Continuous Games 

Again we begin with the definition of this class of games. 

Definition 2.6. A continuous game consists of an arbitrary (possibly infinite) set / of 
players i, each of whom has a pure strategy set Ci which is a compact HausdorfF space and 
a utility function : C — )■ M which is continuous. 

Note that any finite set forms a compact Hausdorff space under the discrete topology 
and any function out of such a set is continuous, so the class of continuous games includes 
the finite games. Another class of continuous games are the polynomial games, which are 
our primary focus when we study computation of correlated equilibria in the sections which 
follow. The theorems and proofs below can safely be read with polynomial games in mind, 
ignoring such topological subtleties as regularity of measures. However the extra generality of 
arbitrary continuous games requires little additional work in the proofs of the characterization 
theorems, so we will not formally restrict our attention to polynomial games here. 

Definition 2.7. A polynomial game is a continuous game with n < oo players in which the 
pure strategy spaces are Ci = [—1,1] for all players and the utility functions are polynomials. 

Defining correlated equilibria in continuous games requires somewhat more care than in 
finite games. Because of the technical difficulties of dealing with conditional distributions 
on continuous spaces, it is preferable not to formulate our new definition by generalizing 



Definition 2.2 directly. An obvious thing to try would be to replace the sum in Proposition 
2.3 with an integral and to choose that as the definition of correlated equilibria in continuous 
games. That would be simple enough, but this leads to a notion which is very weak and 
uninformative. Since we would be integrating over "slices" our candidate definition would be 
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met, for example, by any continuous probability distribution regardless of the game chosen. 
Thus we have to use a different approach. 

The standard definition of correlated equilibria in continuous games (as used in [15J) 



instead follows Proposition |2.5[ In this case we must add the additional assumption that the 
departure functions be BoreQ measurable to ensure that the integrals are defined. For finite 



games this assumption is vacuous so this definition is equivalent to Definition 2.2 



Definition 2.8. A correlated equilibrium of a continuous game is a joint probability 
measure tt G A(C) such that 



N(Ci(si),s_i) -Ui{s)] dTc{s) < 

for all i and all Borel measurable departure functions Q. 

Before stating and proving alternative characterization theorems, we will discuss some of 
the difficulties of working with correlated equilibria in continuous games and this definition 
in particular. The goal here is to motivate the need for alternative characterizations. 

The problem of computing Nash equilibria of polynomial games can be formulated ex- 
actly as a finite-dimensional nonlinear program or as a system of polynomial equations and 
inequalities [21] • The key feature of the problem which makes this possible is the fact that it 
has an explicit finite-dimensional formulation in terms of the moments of the players' mixed 
strategies. 

To see this, suppose that player 1 chooses his action x G [—1, 1] according to a mixed 
strategy a (a probability distribution over [—1, 1]). Each player's utility function is a mul- 
tivariate polynomial which only contains terms whose degree in x is at most some constant 
integer d. Then regardless of how everyone chooses their strategies, their expected utility 
will only depend on a through the moments J x da{x), J da{x), ■ ■ ■ , J x'^ da{x). Therefore 
player 1 can switch from cr to any other mixed strategy with the same first d moments with- 
out affecting game play, and we can think of the Nash equilibrium problem as one in which 
each player seeks to choose moments which correspond to an actual probability distribution 
and form a Nash equilibrium. 

On the other hand there is no exact finite-dimensional characterization of the set of 
correlated equilibria in polynomial games; for a counterexample see [32]. Given the char- 
acterization of Nash equilibria in terms of moments, a natural attempt would be to try to 
characterize correlated equilibria in terms of the joint moments, i.e. the values J s^^--- s,^" dn 
for nonnegative integers ki and joint measures vr. In fact we will be able to obtain such a 
characterization below, albeit in terms of infinitely many joint moments. The reason this 
attempt fails to yield a finite dimensional formulation is that the definition of a correlated 
equilibrium implicitly imposes constraints on the conditional distributions of the equilibrium 
measure. A finite set of moments does not contain enough information about these condi- 
tional distributions to check the required constraints exactly. Therefore we also consider 
approximate correlated equilibria. 



-'^The Borel cr- algebra is the cr- algebra generated by the topology on Ci, which was assumed given in the 
definition of a continuous game. 
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Definition 2.9. An e-correlated equilibrium of a continuous game is a joint probability 
measure vr G A(C) such that 

[uiiCiisi), - Ui{s)] dTr{s) < e 

for all i and all Borel measurable departure functions Q. This definition reduces to that of 
a correlated equilibrium when e = 0. 

That is to say, e-correlated equilibria are distributions of recommendations in which 
no player can improve his expected payoff by more than e by deviating from his recom- 
mendation unilaterally. Compare this definition to the main characterization theorem for 



e-correlated equilibria below (Theorem 2.10). This theorem shows that e-correlated equilib- 
ria can equivalently be defined by integrating the utilities against any sufficiently rich class 
of test functions, instead of by using measurable departure functions. Intuitively, the advan- 
tage of this characterization is that the product fi{si)ui{ti, S-i) is a "simpler" mathematical 
object than the composition Ui{Ci{si), s_i), especially when /j, ti, and d are allowed to vary. 
While this characterization does not have an obvious game theoretic interpretation, it allows 
us to compute correlated equilibria both algorithmically (Section [3]) and analytically [32] . 
There also exist a variety of characterizations in which the departure functions are re- 



stricted to lie in a particular class (e.g.. Lemma 2.11 below and similar results in [33]) and no 
test functions are used. These characterizations have the advantages of conceptual simplic- 
ity and ease of interpretation. However, any characterization involving departure functions 
suffers from the difficulty that compositions of the utilities and the departure functions must 
be computed and these will likely be complex even if the departure functions are restricted 
to a simple class. The difficulty is magnified by the fact that even these restricted classes of 
departure functions are large and often difficult to parametrize in a way which is amenable 
to computation. Therefore it seems that departure function characterizations of correlated 
equilibria cannot be applied directly to yield effective computational procedures. 

Theorem 2.10. A probability measure tt G A(C) is an e-correlated equilibrium of a contin- 
uous game if and only if for all players i, positive integers k, strategies tj,. . . ,t^ G Ci, and 
functions . . . , f^ : Ci [0, 1] in one of the classes 

1. Weighted measurable characteristic functions, 

2. Measurable simple functions (i.e., functions with finite range), 

3. Measurable functions, 

4. Continuous functions, 

5. Squares of polynomials (if Ci C M'^^ for some ki for all i). 
such that Yl'j=ifii^i) — 1 /^'^ ^ inequality 
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holds. 



To prove this, we need several approximation lemmas. 

Lemma 2.11 (A special case of Lemma 20 in [33]). Simple departure functions (those with 
finite range) suffice to define e-correlated equilibria in continuous games. That is to say, a 
joint measure tc is an e-correlated equilibrium if and only if 

j [ui{ii{si),s_i) - Ui{s)] d7r{s) < e 

for all players i and all Borel measurable simple departure functions ^j. 

Proof. The forward direction is trivial. To prove the reverse, first fix i. Then choose any 
measurable departure function Q and let 5 > be arbitrary. By the continuity of Ui and 
compactness of the strategy spaces there exists a finite open cover U^,. . . of Ci such 
that Si, s'i G implies \ui{si, S-i) — Ui^s'^, < 6 for all S-i G C-i and j = 1, . . . ,k. Fix 
any sj G for all j. Define a simple measurable departure function by ^i{si) = sj where 
j = min{/ : (i{si) G U^}. Then \ui{Q{si), S-i) - Ui{^i{si), S-i)\ < 6 for all s G C, so 

j [uiiCiisi), S-i) - Ui{s)] dTx{s) < j S-i) + 5 - Ui{s)] d7r{s) < e + S. 

Letting 6 go to zero completes the proof. □ 

Lemma 2.12. If C is a compact Hausdorff space, fi is a finite regular Borel measure on C , 
. . . , : C — )■ [0, 1] are measurable functions such that Yl'j=i f'' — ^> '^'^'^ 6 > 0, then there 
exist continuous functions g^, . . . , g'' : C ^ [0, 1] such that ^{{x G C : f^{x) ^ g^{x)}) < S 
for all i and Y!j=i9^ < 1- 

Proof. We can apply Lusin's theorem which states exactly this result in the case k = 1 [2^ . 
If /c > 1, then we can apply the k = 1 case with | in place of 5 to each of the fK Call 
the resulting continuous functions gK Then G C : f\x) ^ g^{x) for some j}) < 5. 

But X]j=i /"^ — 1) so ^.{{x G C : Yl^j=i9''{^) > 1}) < ^- Let h{x) = max{l, ^^^^ ^■'(x)} so 
/i : C — )■ [1, oo) is a continuous map. Define g^{x) = ^j^- Then the g^ are continuous, sum 
to at most unity, and are equal to the f^ wherever all of the g^ equal the f^, i.e. except on 
a set of measure at most 5. □ 

Lemma 2.13. If C C M'^ is compact, f^, . . . , : C — t- [0, 1] are continuous functions such 
that Yl^=i f'' — ^' ^'^^ 6 > 0, then there exist polynomials p^, . . . : C — t- [0, 1] which are 
squares such that \ f^{x) — p^{x)\ < 5 for all x E C and J2'j=iP' — ^■ 

Proof. By the Stone- Weierstrass theorem, any continuous function on a compact subset of 
can be approximated by a polynomial arbitrarily well with respect to the sup norm. 
Approximating the square root of a nonnegative function / using this theorem and squaring 
the resulting polynomial shows that a nonnegative continuous function on a compact subset 
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of M.'^ can be approximated arbitrarily well by a square of a polynomial with respect to the 
sup norm. 

Let be a square of a polynomial which approximates within ^ in the sup norm. 
Since takes values in [0, 1], p' takes values in [0, 1 + Let = Then for all 

X E C we have p'lx) < fP{x) and 

p'{x) — p^{x) = p'{x) — ^ ^ ] = 

so p'{x) is within | of p^{x) for all x E C. By the triangle inequality p^ approximates 
within 6 in the sup norm. Furthermore for all a; G C we have 




Proof of Theorem 2.1(\ First we prove that if vr is an e-correlated equilibrium then ([s]) holds 



in the case where the // are simple. We can choose a partition Bj, . . . , Bl of Ci into disjoint 
measurable sets such that // = Y^^^i CjmXBY' where Cjm E [0, 1] and xb^^ denotes the 
indicator function which is unity on and zero elsewhere. Define a departure function 
Ci : Cj — 7- Ci piecewise on the B^ as follows. If 

/ [ui{tl, s-i) - Ui{s)] dir 

is nonnegative for some j define C,i{si) = tj for all Sj E B^ where j is chosen to maximize 
the above integral. If the integral is negative for all j define Ci{si) = s, for all Sj E B™-. Then 
we have 

y^Cjm [ui{tl,S-i) - Ui{s)] dn < / [Ui{Ci{Si),S-i) -Ui{s)] dTT 

j=l Jby'xc^, JbY'xC-, 

for all m. Summing over m and using the definition of an e-correlated equilibrium yields ^ 
in the case where the // are simple. 

Conversely suppose that ^ holds for all measurable simple functions. Let Q : Ci — )■ Ci 
be any simple departure function. Let t\^...^t\ be the range of and Bf = Ci~^({^i})- 
Defining // = x^j , ^ says exactly that vr satisfies the e-correlated equilibrium condition for 



the departure function Q. By Lemma 2.11, n is an e-correlated equilibrium. 

Any simple function can be written as a sum of weighted characteristic functions, so by 
making several of the the same, we see that Q for weighted characteristic functions is 
the same as (|3| for simple measurable functions. If the inequality ^ holds for all simple 
measurable functions, a standard limiting argument proves that it holds for all measurable 
//, hence for all continuous //. 
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Suppose conversely that ([s]) holds for all continuous //. Fix any measurable // sat- 
isfying the assumptions of the theorem. Define a signed measure nl on Ci by {Bi) = 
Ib xc ^-i) ~ "^ii^)] diT- Let /ij = X]j=iki I fix any 5 > 0. Then by the Lemma 

2.12 there exist continuous functions gl : Ci ^ [0, 1] which sum to at most unity and equal 
the // except on a set of /ij measure at most 5. Therefore 

/ fiXsi) [ui{tl,s_i) -Ui{s)\ dir-^ gl{si) [ui{tl , s _i) - Ui{s)\ dn 
i=i j=i 

<^IWisi)-gi{s^\d7rl<2kS, 



so 



.7 = 1 ^ 



uAs)] dn <e + 2kS. 



But 5 was arbitrary, so (|3| holds for all measurable //. 

Finally assume Ci C M^' for some ki. If ^ holds for all continuous //, then it holds for 
all squares of polynomials. Suppose conversely that it holds for all squares of polynomials. 
Let be any continuous functions satisfying the assumptions of the theorem and 6 > 0. Let 
pI be polynomials squares which approximate the // within 6 in the sup norm and satisfy 
the assumptions of the theorem, as provided by Lemma |2.13[ Then 



J fi^^^^ [ui{tl,s.i) -u,{s)\ dn-^j pl{si) [u^{tl,s_i) -Ui{s)] drr 



k fi k fi 

But 5 was arbitrary and the integrals on the right are finite, so (|3| holds for all continuous 
//• □ 



Several simplifications occur when specializing Theorem 2.10 to the e = case, yielding 
the following characterization. We will use the polynomial condition of this corollary in 
Section 3.3 to develop algorithms for computing (approximate) correlated equilibria. The 
characteristic function condition is used to compute extreme correlated equilibria of an ex- 
ample game in 
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Corollary 2.14. A joint measure it is a correlated equilibrium of a continuous game if and 
only if 

fi{si) [ui{ti, s^i) - Ui{s)] d7r{s) < (4) 



for all i and ti G Ci as fi ranges over any of the following sets of functions from Ci to [0, oo) ; 

1. Characteristic functions of measurable sets, 

2. Measurable simple functions, 

3. Bounded measurable functions, 
4- Continuous functions, 

5. Squares of polynomials (if Ci C M'^' for some ki for all i). 

Proof. When e = the k = 1 case of equation ([s]) implies the A; > 1 cases. Furthermore 
e = makes ^ homogeneous, so it is unaffected by positive scaling of the //, which allows 
us to drop the assumption fi<l. □ 



Theorem 2.10 also has important topological implications for the structure of e-correlated 
equilibria. Recall that the weak* topology on the set of probability distributions A(C) over 
a compact Hausdorff space is the weakest topology which makes tt h-> J f dn a. continuous 
functional whenever / : C M is a continuous function. 

Corollary 2.15. The set of e-correlated equilibria of a continuous game is weak* compact. 



Proof. By the continuous test function condition in Theorem 2.10 the set of e-correlated 
equilibria is defined by conditions of the form j f dn < e where / ranges over continuous 
functions of the form ^^=1 fti^i) \ui{t^i-, S-i) — Ui{s)\ . By definition this presents the set of 
e-correlated equilibria as the intersection of a family of weak* closed sets. Hence the set of e- 
correlated equilibria is a closed subset of A(C). But A(C) is compact by the Banach-Alaoglu 
theorem p9], so the set of e-correlated equilibria is compact. □ 

Corollary 2.16. Ifn^ is a sequence of e^- correlated equilibria and — )■ 0, then the sequence 
Tx^ has a weak* limit poin^ and any such limit point is a correlated equilibrium. 

^It is important to note that here we use the term hmit point to refer to a limit point of a sequence, 
which is shghtly different from a hmit point of the underlying set of values which appear in the sequence. 
The difference is essentially that the singleton set {tt} has no limit points (in the sense of, say, |2in. but we 
would like to say that tt is a limit point of the constant sequence tt, tt, tt, . . .. Rigorously, we say that tt is a 
limit point of the sequence tt^, tt^, tt'^, . . . if for any neighborhood U of tt, there are infinitely many indices i 
such that TT* S U . Equivalently (at least in a Hausdorff space) tt is a limit point of the sequence if and only 
if TT appears infinitely often or tt is a limit point of the underlying set {7r'^|fc G N}. 
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Proof. If there is some vr such that n'' = tt for infinitely many k, then vr is a hmit point of the 
sequence. Also vr is an e'^-correlated equilibrium for arbitrarily small e^, so it is a correlated 
equilibrium and we are done. Otherwise, the sequence ir^ contains infinitely many points. 
The space A(C) with the weak* topology is compact by the Banach-Alaoglu theorem [29] . 
hence any infinite set has a limit point. Let vr G A(C) be a limit point of the sequence vr*'. 
For any e > there exists such that for all k > k^, tt'' is an e-correlated equilibrium. The 
set A(C) is Hausdorff [21], so vr is also a limit point of the set {'n'^}k>ko- Since the set of 



e-correlated equilibria is compact by Corollary 2.15 the limit point vr must be an e-correlated 



equilibrium for all e > 0, i.e. a correlated equilibrium. □ 

Finally, we consider e-correlated equilibria which are supported on some finite subset. 
In this case, we obtain another generalization of Proposition |2.3 which we will use in the 



algorithms presented in Section 3.2 



Proposition 2.17. A probability measure tt G A(C'), where C = Uj^jCj is a finite subset 
of C , is an e-correlated equilibrium of a continuous game if and only if there exist ei^si such 
that 

^ ■k{s) [ui{ti, s_i) - Ui{s)] < ei^s, 

s—i^C—i 

for all players i, all Si E Ci, and all ti E Ci, and 



for all players i. 



Proof. If we replace ti with (i{si) in the first inequality then sum over all Sj G Ci and combine 
with the second inequality, we get that 

^ vr(s) [ui{Ci{si), - Ui{s)] < e (5) 
sec 

holds for all i and any function Q : Ci Ci. This is exactly the definition of an e-correlated 
equilibrium in the case when vr is supported on the finite set C. 
Conversely if tt satisfies ([s]) for all Q : Ci ^ Ci then let 

ei^si = max ^ 7r(s) [ui{ti, S-i) - Ui{s)] . 

For each Sj G Q, let C,i{si) be any G Ci which achieves this maximum; such a ti exists 
by compactness and continuity. Substituting this Q into ([s]) shows that tt satisfies the 
assumptions of the theorem. □ 
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3 Computing Correlated Equilibria 



We focus in this section on developing algorithms that can compute approximate correlated 
equilibria with arbitrary accuracy. We consider three types of algorithms, which we will 
illustrate in turn using the example below. 

Example 3.1. Consider the polynomial game with two players, x and y, each choosing their 
strategies from the interval = Cy = [—1,1]. Their utilities are given by 

Ux{x, y) = 0.596x2 ^ 2.072a;?/ - 0.39Ay^ + 1.360a; - 1.200?/ + 0.554 and 
Uy{x,y) = -O.lOSx^ + 1.918x7/ - 1.04%^ - 1.232x + 0.842?/ - 1.886. 

The coefficients have been selected at random. This example is convenient, because as Figure 
[3] shows, the game has a unique correlated equilibrium (the players choose x = y = 1 with 
probability one). For the purposes of visualization and comparison, we will project the 
computed equilibria and approximations thereof into expected utility space, i.e. we will plot 
pairs (J UxdTT,J Uydri^. 



3.1 Static Discretization Methods 

The static discretization methods we present here are slow in practice and should be taken 
as a benchmark against which to compare the methods of later sections. The techniques in 
this section are general enough to apply to arbitrary continuous games with finitely many 
players, so we will not restrict our attention to polynomial games here. 

The basic idea of static discretization methods is to select some finite subset Ci C Cj of 
strategies for each player and limit his strategy choice to that set. Restricting the utility 
functions to the product set C = n"^^C*j produces a finite game, called a sampled game or 
sampled version of the original continuous game. The simplest computational approach is 
then to consider the set of correlated equilibria of this sampled game. This set is defined by 



the linear inequalities in Proposition 2^ along with the conditions that vr be a probability 
measure on C. The complexity of this approach in practice depends on the number of points 
in the discretization. 

The question is then: what kind of approximation does this technique yield? In general 
the correlated equilibria of the sampled game may not have any relation to the set of corre- 
lated equilibria of the original game. The sampled game could, for example, be constructed 
by selecting a single point from each strategy set, in which case the unique probability mea- 
sure over C is automatically a correlated equilibrium of the sampled game but is a correlated 
equilibrium of the original game if and only if the points chosen form a pure strategy Nash 
equilibrium. Nonetheless, it seems intuitively plausible that if a large number of points were 
chosen such that any point of Ci were near a point of Ci then the set of correlated equilibria 
of the finite game would be "close to" the set of correlated equilibria of the original game in 
some sense, despite the fact that each set might contain points not contained in the other. 

To make this precise, we will show how to choose a discretization so that the correlated 
equilibria of the finite game are e-correlated equilibria of the original game. 
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Proposition 3.2. Consider a continuous game with finitely many players, strategy sets Ci, 
and payoffs Ui. For any e > 0, there exists a finite open cover Ul, . . . , Ul' of Ci such that if 
Ci Ci is a finite set chosen to contain at least one point from each Ul, then all correlated 
equilibria of the finite game with strategy spaces Ci and utilities Ui\(j will he e-correlated 
equilibria of the original game. 

Proof. Note that the utihties are continuous functions on a compact set, so for any e > 
we can choose a finite open cover f//, . . . , f/]' such that if Si varies within one of the Uj and 
s_j G C_j is held fixed, the value of Ui changes by no more than e. Let C satisfy the stated 
assumption and let vr be any correlated equilibrium of the corresponding finite game. Then 



by Proposition 2.3 



'K{s)[Ui{ti,S^i) -Ui{s)]<{] 

S — i^C—i 

for all i and all Sj, tj G Ci. Any ti G Ci belongs to the same Ul as some ii & Ci, so 

'r({s)[ui{ti,s-i) - Ui{s)]< ^ 'K{s)[ui[ii,s-i) - Ui{s) + e] <e ^ 7r(s) = e. 



Therefore the assumptions of Proposition 2.17 are satisfied with ej^^. = cX^s.^gC-, ^(■^)- ^ 



Though our primary goal here is to compute correlated equilibria, not prove existence, it is 



worth noting that Proposition 3.2, Corollary 2.16 and the existence of correlated equilibria 



in finite games [15] combine to prove the existence of correlated equilibria in continuous 
games with finitely many players. Indeed, this is proven in \ (along with the extension to 
an arbitrary set of players) with an argument along similar lines. One can view much of the 
contents of the present paper up to this point as expanding on this argument from p[5] . 



The proof of Proposition |3.2| shows that if the utilities are Lipschitz functions, such as 
polynomials, then the Uj can in fact be chosen to be balls with radius proportional to e. If 
the strategy spaces are Ci = [—1,1] as in a polynomial game, then Ci can be chosen to be 

uniformly spaced within [—1,1]. In this case ^ = O (^) where d = maxj Ci . 

Example 3.1 (continued). Figure [l] is a sequence of static discretizations for this game for 
increasing values of d, where d is the number of points in Cx and Cy. These points are 
selected by dividing [—1,1] into d sub intervals of equal length and letting C^ = Cy be the 
set of midpoints of these subintervals. For this game it is possible to show that the rate of 
convergence is in fact © (^) so the worst case bound on convergence rate is achieved in this 
example. 

In fact we can improve this convergence rate to e = O (^) if we include the endpoints 
±1 in Cj as well and assume that the utilities have bounded second derivatives. We omit 
the proof for brevity. 
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static Discretization Correlated Equilibrium Payoffs for d = 10 to 50 (d = strategies / player) 
-1.48 r 



-1.5 



-1.52 



-1.54 
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Exact correlated 
equilibrium payoff 
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d = 45/ 



^ d = 35 X 
d = 30 

d = 25 ^ ^ 
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u 
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Figure 1: Computing a sequence of e-correlated equilibria of the game in Example |3.1 by 



static discretization. Each point represents the (unique) correlated equilibrium of the finite 
game where players are restricted to strategies chosen from a finite set of d strategies equally 
spaced in [—1,1]. The axes represent the utilities received by players x and y. It can be 
shown that the convergence in this example happens at a rate e = © (^)- This is slow 
enough that it is not obvious visually that the sequence of points is converging to the exact 
correlated equilibrium payoff, though we can prove that it is (e.g., by combining Proposition 



3.2 with Figure k3l which shows that the equilibrium payoff is unique). 
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3.2 Adaptive Discretization Methods 

3.2.1 A family of convergent adaptive discretization algorithms 

In this section we consider continuous games with finitely many players and provide two 
algorithms (the second is in fact a parametrized family of algorithms which generalizes 
the first) to compute a sequence of e'^-correlated equilibria such that lim/;_>oo = 0. By 
Corollary |2. 16 any limit point of this sequence is a correlated equilibrium. We will show that 



for polynomial games these algorithms can be implemented efficiently using semidefinite 
programming. 

Informally, these algorithms work as follows. Each iteration k begins with a finite set 
Cf ^ Ci of strategies which each player i is allowed to play with positive probability in that 
iteration; the initial choice of this set at iteration = is arbitrary. We then compute the 
"best" e-correlated equilibrium in which players are restricted to use only these strategies, 
i.e., the one which minimizes e (subject to some extra technical conditions needed to ensure 
convergence). 

Given the optimal objective value and optimal probability distribution tt'^, there is some 
player i who can improve his payoff by if he switches from his recommended strategies to 
certain other strategies. We interpret these other strategies as good choices for that player 
to use to help make smaller in later iterations k. Therefore we add these strategies to 
to get C^^^ and repeat this process for iteration k + 1. 

Algorithm 3.3. Fix a continuous game with finitely many players. Let A; = and for each 
player fix a finite subset C d. 

• Let be an e'^-correlated equilibrium of the game having minimal subject to two 
extra conditions. First, tt^ must be supported on . Second, we require that vr'^ be 
an exact correlated equilibrium of the finite game induced when deviations from the 
recommended strategi es are restricted to the set , i.e. when we replace the condition 



ti G Ci in Proposition 2.17 with tj G . 

That is to say, let be the optimal value of the following optimization problem, and 

TT^ be an optimal assignment to the decision variables. 

minimize e 

subject to 

y^ Tx{s) [ui{ti, S-i) — Ui{s)] < for all i and Si,ti G 

7r(s) [ui{ti, S-i) - Ui{s)] < Ci^s, for aU i, Si G and U e Q 

y] ei^Si < e for all i 

7r(s) > for all s G (7 

s&C^ 
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• If e'^ = 0, terminate. 

• For each player i for whom Xlsiec* ^'''^i ~ ^' ^'^^^ C^^^ from by adding in, for each 
Si G such that e^^s^ > 0, at least one strategy which makes 

7r(s) [ui{ti, s-i) - Ui{s)] = ei^s,- 

• For all other players i, let C'f'^^ = Cf. 

• Let A; = + 1 and repeat. 

Note that all steps of this algorithm are well-defined. First, the optimization problem 
is feasible. To see this let vr^ be any exact correlated equilibrium of the finite game with 
strategy spaces and utilities Ui restricted to C^; such an equilibrium exists because all 
finite games have correlated equilibria [15]. The Ui are bounded on C (being continuous 
functions on a compact set), so by making e and the ei^s, large, we see that vr*"' is a feasible 
solution of the problem. 

Second, the optimal objective value is achieved by some because the space of prob- 
ability measures on is compact, the constraints are closed, and e is bounded below by 
zero. 

Third, the set of new strategies added in the third bullet is nonempty. Suppose for a 
contradiction that this set were empty for each i such that J^Si^c'' ^h^i = e and each Sj G 
such that ei^s, > 0. By continuity of Ut and compactness of Cj, the left-hand side of the 
e-correlated equilibrium constraint achieves its maximum as a function of ti G Cj. If this 
maximum value were less than ej^^., then the value of ei^si could be decreased. If this could 
be done for all i such that X^SiSC'^ ~ ^ then e itself could be decreased, contradicting 
optimality of tt'^. 

Fourth, this set of new strategies added in the third bullet consists only of strategies which 
are not in because we have the constraint that the deviations in utility be nonpositive 
for h G &. 



To show that Algorithm 3^ converges, we will view it as a member of the following 
family of algorithms with the parameters set to a = and /3 = 1. Varying these parameters 
corresponds to adding some slack in the exact correlated equilibrium constraints and allowing 
some degree of suboptimahty in the choice of strategies added to to form C^~^^. Such 
changes make little conceptual difference, but could be helpful in practice by making the 
optimization problem strictly feasible and allowing it to be solved to within a known fraction 
of the optimal objective value rather than all the way to optimality. We will prove that all 
algorithms in this family converge, that is, with these algorithms converges to zero in the 
limit. 

Algorithm 3.4. Fix a continuous game with finitely many players and parameters < a < 
P < 1. Let k = and for each player fix a finite subset C d. 
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• Choose to be the smallest number for which there exists ir^ such that: 



— TT^ is a probability distribution supported on C'^, 

— TT^ is an e'^-correlated equilibrium of the game, 

— 71^ is not an e-correlated equilibrium for any e < e^, 

— vr'^ is an ae'^-correlated equilibrium of the game when strategy deviations are re- 
stricted to (i.e., when the condition tj G Cj is changed to ti G in Proposition 
2A7\ . 

• If e'^ = 0, terminate. 

• For at least one value of i, form C^^^ from by adding strategies tj^^. G Cj such that 

sec'' 



For all other values of i, let C^^^ = C^. 
Let k = k + 1 and repeat. 



Proposition 3.5. The steps of Algorithm 3.4 are well-defined. 



Proof. It is not immediately obvious that the first step of the algorithm is well-defined, i.e., 
that a minimal e'^ (or any e'^ for that matter) satisfying these conditions exists. To see this let 
ir^'^ be an exact correlated equilibrium of the finite game induced when strategy deviations 
are restricted to C^, and let e'''^ > be the smallest value such that vr'^'^ is an e'^'^-correlated 
equilibrium. Then the pair (vr*^'^, e'^'^) satisfies the four conditions under the first bullet 
above. This shows that the set of values satisfying these conditions is nonempty. 

Choose some sequence (vr'^'', e'^''), / = 1,2,..., of pairs satisfying these conditions such 
that the limit e'' = lim^^oo e'^'' is the infimum over all values of pairs satisfying these 
conditions. Passing to a subsequence if necessary we can assume without loss of generality 



that the n^'^ converge to some n^. It is clear from the proof of Corollary 2.16 that vr'^ is 
an e^-correlated equilibrium supported on which is an ae^-correlated equilibrium when 
deviations are restric ted t o C^. 

From Proposition 2.17 we see that for a fixed support C^, the minimal value of e for which 



a probability measure vr on is a correlated equilibrium of the game varies continuously 
with the probabilities 7t{s) for s & . Therefore is not an e-correlated equilibrium for 
any e < e^. Note that this final step depends crucially on the fact that is finite and 
fixed while / varies. Also note that this subtlety disappears if a = because in that case 
it wouldn't matter if the limiting distribution had a smaller e value. It is clear that the 
remaining steps of the algorithm are well-defined. □ 



Theorem 3.6. Fix a continuous game with finitely many players. Algorithms 3^ and 3.4 



converge to the set of correlated equilibria, i.e., they converge in the sense that e 0. 
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Proof. Suppose not, so there exists e > and infinitely many values of k such that > e. For 
each i let Bl, . . . , B^' be a finite open cover of such that Ui{si, S-i) —Ui{ti, < ^{P — a)e 
when Si and ti belong to the same set Bl and s_j G C_j. Such a cover exists by the 
compactness of the Ci and the continuity of the Ui. There are finitely many sets 5- so there 
is some iteration k, which we can take to satisfy > e, such that for all i all of the sets Bl 
which will ever contain an element of at some iteration k already do. 

Note that tt^ is an ae'^-correlated equilibrium when strategy choices are restricted to C^, 
and e'^ > so we have jSe'' > ae^. By the minimality of e^, the set C^^^ \ Cf is nonempty for 
some player i (that is to say, it is always possible to perform the third step of the algorithm). 
Choose such an i and ti^si ^ C^^^ which satisfy 



■K^{s) [Ui{ti 



Ui{s)] > (3eK 



By assumption, for any choice of rj G we have 

^ n''{s) [ui{ri^s,,s^i) -Ui{s)] < ae^ , 
sec'' 



so 



By construction of k, we can choose rj^^. G to lie in the same set Bl as tj^^. for each 
s,- G Thus 



< 



sec'' 



sec* 



a contradiction. 



□ 



Now we will illustrate Algorithm |3.3| on two examples. 



3.3 



initialized with = a° 



Example 3.1 (continued). In Figure |2j we illustrate Algorithm 
{0}. In this case convergence is obtained in three iterations, significantly faster than the 
static discretization method. The resulting strategy sets were = = {0, 1}. 
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Adaptive Discretization e-Correlated Equilibrium Payofts 
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Figure 2: Convergence of Algorithm 3.3 (note the change in scale from Figure [T]). At each 
iteration, the expected utility pair is plotted along with the computed value of e for which 
that iterate is an e-correlated equilibrium of the game. In this case convergence to e = (to 
within numerical error) occurred in three iterations. 
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0.99 


{0} 


{0} 


{0} 


1 


4.16 






{0.89} 


2 


5.76 


{-1} 






3 


0.57 




{1} 




4 


0.28 


{0.53} 




{0.50,0.63} 


5 


0.16 




{0.49,0.70} 




6 


10-^ 




{-1,0.60} 


{-0.60,0.47} 



Table 2: Output of Algorithm 3.3 on a three player polynomial game with utilities of degree 
4 and randomly chosen coefficients. 



Example 3.7. For a more complex illustration, we consider a polynomial game with three 
players, choosing strategies x, and z G [—1, 1]. The utilities were chosen to be polynomials 
with terms up to degree 4 in all the variables and the coefficients were chosen independently 
according to a normal distribution with zero mean and unit variance (their actual values are 

proceeds as in Table [2| which shows the value of and the new 
strategies added on each iteration. The terminal probability distribution tt^ does not display 
any obvious structure; in particular it is not a Nash equilibrium (product distribution). 



omitted). Algorithm 3.3 



3.2.2 Implementing these algorithms with semidefinite programs 

To implement these algorithms for polynomial games, we must be able to do two things. 
First, we need to solve optimization problems with finitely many decision variables, linear 
objective functions and two types of constraints: nonnegativity constraints on linear func- 
tionals of the decision variables, and nonnegativity constraints on univariate polynomials 
whose coefficients are linear functionals of the decision variables. That is to say, we must be 
able to handle constraints of the form p{t) > for all t G [—1, 1], where the coefficients of 
the polynomial p are linear in the decision variables. Second, we need to extract values of t 
for which such polynomial inequalities are tight at the optimum. 

Both of these tasks can be done simultaneously by casting the problem as a semidefinite 
program (SDP). For an overview of semidefinite programs and a summary of the necessary 
results (both of which are classical), see the appendix. 



In the optimization problem in Algorithm 3.3 we have a finite number of univariate 
polynomials in whose coefficients are linear in the decision variables 7i{s) and ej^g.. We 
wish to constrain these coefficients to allow only polynomials which are nonnegative for all 



ti G [—1, 1]. By Propositions A. 3 and A. 4 in the appendix this is the same as asking that 
these coefficients equal certain linear functions of matrices (i.e., sums along antidiagonals) 
which are constrained to be symmetric and positive semidefinite. Therefore we can write 
this optimization problem as a semidefinite program. 

As a special case of convex programs, semidefinite programs have a rich duality theory 
which is useful for theoretical and computational purposes. In particular, SDP solvers keep 
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a 


b 


c 


a 





1 





b 


1 


5 


7 


c 





7 






Table 3: A finite symmetric game with identical utilities for which Algorithm 3.4 with 
a = (3 = 1 does not converge when started with strategy sets (7° = = {a}. 



track of feasible primal and dual solutions in order to determine when optimality is reached. 
It can be shown that the dual data obtained by an SDP solver run on this optimization 
problem will encode the values of tj making the polynomial inequalities tight at the optimum 
|24j. 

The process of generating an SDP from the optimization problem in the algorithms above, 
solving it, and extracting an optimal solution along with tj values from the dual can all be 
automated. We have done so using the SOSTOOLS MATLAB toolbox for the pre- and 
post-processing and SeDuMi for solving the semidefinite programs efficiently [26l IM] . 

3.2.3 A nonconvergent limiting case 

Note that in the algorithms above the convergence of the sequence is not necessarily 
monotone. If we were to let a = (3 (a case we did not allow above), the sequence would 
become monotone nonincreasing. If we were to furthermore fix a = /3 = 1, then the condition 
that vr be an exact (or ae'^-) correlated equilibrium when deviations are restricted to Cf would 
become redundant and could be removed. 



These changes would simplify the behavior of Algorithm 3.4 conceptually as well as 
reducing the size of the SDP solved at each iteration, so we would like to adopt them if 
possible. However, the resulting algorithm may not converge, in the sense that may 
remain bounded away from zero. 

Example 3.8. Consider the game shown in Table |3j which is symmetric and has identical 



utilities for both players. Let Cj' = = {a} and apply Algorithm 3.3, but remove the 
condition that tt'^ be an exact correlated equilibrium when deviations are restricted to C^. 
The only probability distribution supported on is 5(^a,a) which has an objective value of 
e° = 1. It is easy to see that C} is formed by simply adding each player's best response to 
a, so that C\ = = {a, 6}. We will argue that the unique solution to the optimization 
problem in iteration /c = 1 is also 5(a,a), hence = C} and the algorithm gets "stuck", so 
that e'' = e° = 1 for all k. 

For a probability distribution vr, let vr^ denote vr with the players interchanged. By 
symmetry and convexity, if vr is an optimal solution then so is "^^^ , which is a symmetric 
probability distribution with respect to the two players. Hence an optimal solution which 
is symmetric always exists. We will parametrize such distributions by vr = p5{a,a) + <i5{a,b) + 
l^ib^a) + f^{h,b)) where J9, g, r > and p + 2q + r = 1. Define a departure function ( : Ci —> Ci 
by ({a) = b, ({b) = C(c) = c. Then for vr to be an e-correlated equilibrium it must satisfy 
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{1} 



Table 4: Output of Algorithm 3.3 for a polynomial game on which Algorithm 3.4 with 
a = /3 = 1 does not converge to a correlated equilibrium. 



the following condition: 



> 



siec-i sec 



^ 7r(s) [mi(C(si), S2) - S2)] 



p + 4g — g + 2r = l + g + r. 



We know we can achieve e = 1 with p 



I.e. TT 



TT 



^(a.a)), and this inequality 



shows that if p < 1 then e > 1. Therefore the minimal e value in iteration A; = 1 is 
unity and is achieved by vr = 5, 



(a,a)- Furthermore we have shown that this is the unique 
symmetric probability distribution which achieves the minimal value of e. Hence any other 
(not necessarily symmetric) optimal solution vr satisfies '^^^ = 5(a,a)- But S(^a,a) is an extreme 
point of the convex set of probability distributions on C^, so we must in fact have n = 5(a,a)- 
Therefore vr^ = vr*^ = S(^a,a) is the unique optimal solution on iteration = 1, so the procedure 
must get stuck as claimed. That is, = {a, b} and = 1 for all k > 1. 

The same behavior can occur in polynomial games, as can be shown by "embedding" the 
above finite game in a polynomial game. For example, we can take = Cy = [—1, 1] and 

Ux{,x, y) = Uy{x, y) = {1- x^)(3y^ + Qy + 5) 
+ (1 -?/2)(3x2 + 6x + 5). 

{—1} the same analysis as above shows that 



Then if C2 

e 



^ -2{oi all A; > 1. 



(^y 



{-1,0} and 



Example 3.9. If we run Algorithm |3.3| on this polynomial game, the iterations proceed as in 
Table |4| The correlated equilibrium obtained in iteration 2 is 

.2 



TT 



0.49225(a; = 0,y = l) + 0.4922(5(x = l,y 
+ 0.01565(x = l,y = 1), 



0) 



i.e., a probability of 0.4922 is assigned to each of the outcomes {x,y) = (0, 1) and {x,y) 
(1,0) and a probability of 0.0156 is assigned to {x,y) = (1, 1). 



3.3 Moment Relaxation Methods 

In this section we again consider only polynomial games. The moment relaxation meth- 
ods for computing correlated equilibria have a different flavor from the discretization meth- 
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ods discussed above. Instead of using tractable finite approximations of the correlated equi- 
librium problem derived via discretizations, we begin with the alternative exact characteri- 



zation given in condition |5] of Corollary |2.14[ In particular, a measure tt on C is a correlated 
equilibrium if and only if 



P^{si) [ui{ti, s-i) - Ui{s)] d7r{s) < 



(6) 



for all i, ti G Cj, and polynomials p. If we wish to check all these conditions for polynomials 
p of degree less than or equal to d, we can form the matrices 



^d+2 



^2d 



Let c be a column vector of length d + 1 whose entries are the coefficients of p, so p'^{si) 



d Sfc. If we define 



Sf [Ui(ti, - Ui{s)] d7!-{s), 



pd+1 



then ^ is satisfied for all p of degree at most d if and only if c' Mf{ti)c < for all c G 
and ti G Cj, i.e. if and only if Mf{ti) is negative semidefinite for all tj G Cj. 

The matrix Mf{ti) has entries which are polynomials in ti with coefficients which are 
linear in the joint moments of tt. By Proposition A. 5 in the appendix, Mf(ti) is negative 
semidefinite for all ti G [—1,1] for a given d and a fixed vr if and only if there exists a 
certificate of a certain form proving this condition holds. We can write a semidefinite program 



again, see Proposition A. 5 in the appendix) in which the decision variables represent such a 



certificate, so we can check this condition by solving the semidefinite program. As d increases 
we obtain a sequence of semidefinite relaxations of the correlated equilibrium problem and 
these converge to the exact condition for a correlated equilibrium. That is to say, for a 
measure to be a correlated equilibrium it is necessary and sufficient that its moments be 
feasible for all of these semidefinite programs. 

We can also let the measure tt vary by replacing the moments of vr with variables and 
constraining these variables to satisfy some necessary conditions for the moments of a joint 
measure on C (see appendix). These conditions can be expressed in terms of semidefinite 
constraints and there is a sequence of these conditions which converges to a description 
of the exact set of moments of a joint measure vr. Thus we obtain a nested sequence of 
semidefinite relaxations of the set of moments of measures which are correlated equilibria, 
and this sequence converges to the set of correlated equilibria. 

Example 3.1 (continued). Figure [s] shows moment relaxations of orders d = 0, 1, and 2. Since 
moment relaxations are outer approximations to the set of correlated equilibria (having been 
defined by necessary conditions which correlated equilibria must satisfy) and the 2"^^ order 



26 



Moment Relaxation Correlated Equilibrium Payoffs of Order d = 0, 1 , and 2 
-0.6 - 
-0.8 - 

-1 
-1.2 
-1.4 
-1.6 
-1.8 
-2 
-2.2 
-2.4 
-2.6 




r'orde 



(unique payoffs) 



0.5 



1.5 



2.5 



Figure 3: Semidefinite relaxations approximating the set of correlated equilibrium payoffs. 
The second order relaxation is a singleton, so this game has a unique correlated equilibrium 
payoff (and in fact a unique correlated equihbrium). 



moment relaxation corresponds to a unique point in expected utility space, all correlated 
equilibria of the example game have exactly this expected utility. In fact, the set of points in 
this relaxation is a singleton (even before being projected into utility space), so this proves 
that this example game has a unique correlated equilibrium. 



4 Future Work 

These results leave several open questions. For any continuous game, the set of correlated 
equilibria is nonempty, and this can be proven constructively as in [15] . Under the same as- 
sumptions we can prove the existence of a Nash equilibrium, but the proof is nonconstructive, 
or at least does not seem to give an efficient algorithm for constructing an equilibrium [21] . In 
the case of polynomial games, existence of a Nash equilibrium immediately gives existence of 
a finitely supported Nash equilibrium by Caratheodory's theorem, which is constructive [31] . 
Therefore there exists a finitely supported correlated equilibrium of any polynomial game. Is 
there a constructive way to prove this fact directly, without going through Nash equilibria? 
Such a proof could potentially lead to a provably efficient algorithm for computing a sample 
correlated equilibrium of a polynomial game. 

While the adaptive discretization and moment relaxation algorithms converge in general 
and work well in practice, we do not know of any results regarding rate of convergence. If 
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we regard the probability distributions produced by these algorithms at the k iteration as 
e'^-correlated equilibria, how fast does e'^ converge to zero? 

Finally, we note that we have merely shown that the adaptive discretization algorithm 
converges to the set of correlated equilibria, not to a particular correlated equilibrium (of 
course it will do so along some subsequence by compactness). Could the algorithm be 
modified to converge to a single correlated equilbrium? Or even better, could one assure 
convergence to a correlated equilibrium with some desirable properties, such as one which 
maximizes the social welfare or (in the polynomial case) is finitely supported? This seems 
plausible given that the algorithm is itself optimization-based, but these problems remain 
open. 
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A Semidefinite programming, sums of squares, and 
moments of measures 

Definition A.l. A semidefinite program is an optimization problem of the form: 

minimize L{S) 
subject to T{S) = v 

5* is a symmetric matrix 

5* ^ (positive semidefinite), 

where L is a given linear functional, T is a given linear transformation, f is a given vector, 
and S" is a square matrix of decision variables. 

Semidefinite programs are convex optimization problems and generalize linear programs 
(T and v can be designed to make S diagonal, in which case the condition S" ^ is the same 
as the condition that S > elementwise) . The solution set of a semidefinite program need 
not be polyhedral, allowing for much more flexibility in modeling than can be achieved with 
linear programs. Many problems can be expressed exactly or approximately as semidefinite 
programs, and this is important because semidefinite programs can be solved efficiently by 
interior point methods. For details and a variety of examples see [35] and [23] . 

The square of a real-valued function is nonnegative on its entire domain, as is a sum of 
squares of real- valued functions. In particular, any polynomial of the form p{x) = ^^pHx), 
where pk are polynomials, is guaranteed to be nonnegative for all x. This gives a sufficient 
condition for a polynomial to be nonnegative. It is a classical result that this condition is 
also necessary if p is univariate [27] . 
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Proposition A. 2. A univariate polynomial p is nonnegative on M if and only if it is a sum 
of squares. 

Proof. A simpler version of the proof of the following proposition. □ 

Proposition A. 3 (Markov-Lukacs [E]). A univariate polynomial p{x) is nonnegative on 
the interval [—1, 1] if and only if p{x) = s{x) + (1 — x^)t(x) where s and t are both sums of 
squares of polynomials. 

Proof. Direct algebraic manipulations show that the set of polynomials of the form s(x) + (l — 
x'^)t{x) where s and t are sums of squares of polynomials in x is closed under multiplication 
and contains all polynomials of the following forms: a for a > 0, (a: — of + 6^ for a, 6 G M, 
X — a for a < —1, and a — x for a > 1. By assumption p{x) factors as a product of terms of 
these types, because any real root of p in the interval (—1, 1) must have even multiplicity. □ 

These sum of squares conditions are easy to express using linear equations and semidefi- 
nite constraints. 

Proposition A. 4. A univariate polynomial p{x) = Yll—oPk^'' of degree at most 2d is a sum 
of squares of polynomials if and only if there exists a symmetric positive semidefinite matrix 
Q E ]R('^+i)^('^+i) such that pk = Yli+j=kQij (numbering the rows and columns of Q from 
to d). 

Proof. Relating the coefficients of p{x) to the entries of Q in this way is the same as writing 
p{x) = x-^Qx where x = [l x x"^ ■ ■ ■ x*^] . Thought of in this way, saying that p{x) is 
a sum of squares is the same as saying that Q = qiqf for some column vectors and 
in this case Q is clearly positive semidefinite. Conversely, if Q is positive semidefinite then 
there exists a matrix F such that Q = F'^F, so p{x) = x^Qx = • ^ 

Similar semidefinite characterizations exist for multivariate polynomials to be sums of 
squares. While the condition of being a sum of squares does not characterize general non- 
negative multivariate polynomials exactly, there exist sequences of sum of squares relaxations 
which can approximate the set of nonnegative polynomials (on e.g. M'^, [—1, l]'^, or a more 
general semialgebraic set) arbitrarily tightly [27] • Furthermore, for some special classes of 
multivariate polynomials, the sum of squares condition is exact. 

Proposition A. 5. A matrix M{t) whose entries are univariate polynomials in t is positive 
semidefinite on [—1, 1] if and only if x'M{t)x = S{x,t) + (1 — t'^)T{x,t) where S and T are 
polynomials which are sums of squares. 

Proof. Follows from Theorem 5.6 of [5]. □ 

Now suppose we wish to answer the question of whether a finite sequence (/i", . . . , fi^) of 
reals correspond to the moments of a measure on [—1,1], i.e. whether there exists a positive 
measure fi on [—1,1] such that fi'' = J x^dfi{x). Clearly if such a measure exists then we 
must have / p{x) d^{x) > for any polynomial p of degree at most k which is nonnegative 
on [—1, 1]. This necessary condition for moments to correspond to a measure turns out to 
be sufficient [17] and can be written in terms of semidefinite constraints. 
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Proposition A. 6. The condition that a finite sequence of numbers {//', . . . , /J') be the mo- 
ments of a positive measure on[—l, 1] can be written in terms of linear equations and semidef- 
inite matrix constraints. 

One can formulate similar questions about whether a finite sequence of numbers cor- 
responds to the joint moments / x^^ ' ' ' dfi{x) of a positive measure ^ on [—1, l]'^ (or a 
more general scmialgcbraic set). Using a sequence of semidefinitc relaxations of the set of 
nonnegative polynomials on [—1, l]'^, a sequence of necessary conditions for joint moments 
is obtained. These conditions approximate the set of joint moments arbitrarily closely. 
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